This invention relates to machine vision, and particularly to methods and apparatuses for image analysis.
Machine vision is often used for inspection, such as inspecting the functioning of various configurations of liquid crystal display (xe2x80x9cLCDxe2x80x9d) elements within displays. Each LCD element has at least two states: on or off. Multiple LCD elements are arranged to create more than one configuration within a display, such as clock displays where the configurations include the numbers zero through nine. Various machine vision algorithms have been applied to inspect LCD displays, including pattern recognition and analysis of image subtraction results.
The pattern-recognition systems generate a template of a LCD configuration and optionally a reference site, such as a set of fiducials. The template, that includes both the LCD configuration and the fiducials, is a specific pattern of grey values in an image. A digital image consists of individual pixels, of various light intensities, having x and y integer coordinates. An image is searched to find a subset of the image that is most similar to the template using one of various search strategies known in the art. A score representing the similarity is produced and used to pass or fail the LCD configuration based on the tolerances provided by a user.
One problem with the pattern-recognition technique as applied to LCD displays is the lack of specificity of the results. The results indicate whether the entire configuration of the template functions, but not whether each LCD element within the configuration is functioning properly.
The image-subtraction technique involves creating a difference image. Again, first a template of a LCD configuration is created. Thereafter, an inspection image is acquired of the LCD elements in the configuration that is being inspected. The inspection image and the template are registered, where registration of a pair of objects shall be defined as orienting a first object with respect to a second object to make all alignment parameters of the first object substantially equal to the corresponding alignment parameters of the second object. For example, a first semiconductor fabrication mask is said to be registered with an overlaying second semiconductor fabrication mask when a pair of fiducial marks of the first mask is located at the same projective position as the corresponding fiducial marks on the second mask. After registration, the inspection image is subtracted from the template to provide the difference image. The non-zero values in the difference image indicate potential errors, which are then analyzed.
There are several problems encountered with analyzing the difference image. One problem is that improper registration of the template to the inspection image creates areas of non-zero intensity in the difference image that appear as potential errors. For example, the boundaries of a feature will appear in the difference image as non-zero values when the template is offset in any direction from the inspection image. Another problem is that areas of non-zero intensity in the difference image can appear large enough to be a fatal defect, which will fail the part, when they are not large enough. For example, if a user specifies a part fails for any defect: more than six pixels in area within a LCD element; a six-pixel error that spans two LCD elements should not fail the part. However, in the difference image a six-pixel error within one LCD element is indistinguishable from a six-pixel error that spans two LCD elements. A further problem is that changes in intensity between the template and the inspection image appear as errors in the difference image even when the inspection image contains no flaws. A still further problem is that if the inspection image contains non-uniform changes in intensity, image subtraction will not work well, if at all. Further still, normal acceptable variations of the elements from part to part may appear as false errors in the difference image even if the variation of the part is within acceptable tolerances. Another cause of potential false error in the difference image is because of features on, or near, the elements. For instance, bubbles in a translucent cover, commonly positioned over the elements, will appear as errors in the difference image. Any of these false errors may indicate the LCD elements are not functioning properly: when the LCD elements are functioning properly. Accordingly, more parts are rejected than necessary. To reduce false errors, the template and the inspection image need to be clean, clear, and have similar intensities. This is often not practical, or it is difficult to achieve.
The invention provides methods and apparatuses for determining the state of elements in an image. An image of the elements is acquired and generally aligned with the image pixels, where the elements each have a state. An element characteristic, that approximately represents each element, is extracted from each element and processed in conjunction with element characteristics of other elements to provide at least one comparison criteria. A threshold value of the comparison criteria is chosen. The threshold value is then used to determine the state of selected ones of the elements.
The element characteristics are processed by comparing differences and/or similarities among the elements globally, locally, and/or among neighbors, including differences of intensity or texture, for example, such as a difference value, or a gradient of the change in intensity, for example.
The threshold value is chosen from the difference values, the intensity values, or the gradient values, for example, previously provided. In one embodiment, the threshold value separates similar elements from dissimilar elements and/or similar element-neighbor pairs from dissimilar element-neighbor pairs.
A preferred embodiment of the methods and apparatuses of the invention is determining the state of display elements, such as LCD elements. In a preferred embodiment, the state of each LCD element is on or off. The LCD elements neighbor each other and are arranged in a matrix. An image is acquired of the matrix of LCD elements each in their own state. The intensity (element characteristic) of approximately each LCD element is extracted from the image, where the intensity value can incorporate portions of the image outside each LCD element depending upon the degree to which positioning of each LCD element is known. Therefore, the intensity value abstracted is said to approximately represent each LCD element. In a preferred embodiment, before extraction, the LCD element is aligned with the image pixels by an Affine Transformation to minimize any error. Thereafter, the intensities of neighboring LCD elements are processed so as to provide a plurality of comparison criteria, where each of the comparison criteria represent selected pairs of neighboring LCD elements. A threshold value is chosen from the comparison criteria and used to determine which LCD elements in the matrix are on or off, where the state of each element is indicated by evaluating at least one comparison criteria derived from the element and the threshold value.
In further aspects, additional comparisons are used alone, or in conjunction with, comparisons to the threshold value to determine state of each of the elements. Additional comparisons of the similarity of each element to a local and/or global average intensity, texture, or other element characteristic are described.
In further aspects, the elements are inspected. In one embodiment, the elements are inspected by comparing a model of the elements to the image of the elements, where the model contains model elements and the image contains test elements. The model is generated, and it contains model elements in substantially the same pattern as the pattern of the test elements, where each of the model elements has a model state. The test elements are associated with a model state of the correspondingly positioned model element. In one embodiment, the correspondingly positioned model element is the model element having the same position as the test element if the model was aligned to the test image. Lastly, the test state is compared against the model state, where the test state is determined as previously described.
In another embodiment, the model is created using the steps previously described of acquiring the image, aligning the image, extracting the element characteristic, processing the element characteristic, choosing a threshold value of the comparison criteria, and determining the model state using the comparison criteria.
In a further embodiment, after the model is generated, the element characteristic of each test element is extracted as one of the element characteristics of a plurality of positions, where the one chosen most closely represents the model state from among the plurality of positions.
In a still further embodiment, after the model is generated, the threshold value for the test elements is determined using the model state information from the model. More particularly, a first histogram is created that contains the comparison criteria of neighboring test elements that are suppose to have different model states and a second histogram is created that contains comparison criteria of neighboring test elements that are suppose to have the same model state. The magnitude of each bin of the first histogram is added to a score graph also having bins, while the magnitude of each bin of the second histogram is subtracted from the score graph for each possible value of the comparison criteria on the histograms. From the score graph, a threshold value of the comparison criteria is chosen that received the highest score.
In a further embodiment, the image intensity is normalized before extracting the element characteristics, where normalizing the intensity of an image, optimally, equalizes the grey values in an image that represent a single brightness. The normalization of the invention minimizes the effect of extraneous variations in image brightness and contrast, whether the extraneous brightness variations are linear or non-linear brightness changes. An extraneous variation in brightness changes is due to factors other than features of the scene to be images. A linear brightness,change in, intensity is one wherein the grey-level value of each pixel varies uniformly across the entire image, and a non-linear brightness change in intensity is a non-uniform change in intensity across an image, which means the grey-level values in one region may vary more than in other regions (i.e. the grey-level values of a object of one brightness varies from one region to another region).
The normalization of the invention makes the grey values much more equal than the various grey values that represented an object of a single brightness in the image before normalization, although it may not equalize all the grey values representing each brightness. Further, the normalization of the invention brings the desired range of grey values and the actual range of grey values much closer, even though it may not exactly equalize the range of grey values (i.e. contrast) between two images or an image and a user""s criteria.
The invention recognizes, among other things, that the positions of the LCD elements are better defined by determining a characteristic for a plurality of possible positions for the LCD elements and choosing the positions having the characteristics closest to the model characteristics, such as intensity value, expected in that position. Defining position more precisely minimizes errors, such as errors caused by boundaries from improper alignment during comparison of the model to the image.
The invention recognizes, among other things, that by processing the element characteristics of neighboring and/or local elements, such as taking differences between neighbors, local threshold values, or local averages, the state of the elements can be determined.
The invention solves several of the problems of the prior art, such as each LCD element can be inspected, boundaries do not appreciably contribute to false errors, and the effect of changes in intensity, local and or global, between the model and the run-time image is minimized.
In further aspects, the invention provides an apparatus in accord with the methods described above. The aforementioned and other aspects of the invention are evident in the drawings and in the description that follows.